Systems and methods for selecting content

ABSTRACT

Systems and methods of the disclosure relate to selecting content via a computer network. The system can receive a query to generate content selection criteria. The system can identify an entity of the query and a query graph based on the entity. The system can access a database to identify a template corresponding to the query graph. The template can include a topology and a named variable. The system can determine multiple semantic criteria corresponding to the named variable that match the query graph. The system can use a statistical metric of each of the matching semantic criteria to select candidate content selection criteria.

BACKGROUND

In a networked environment such as the Internet, web publishers such aspeople or companies can provide information for display on web pages orother documents. The web pages can include text, video, or audioinformation provided by the entities via a web page server for displayon the internet. Content providers, such as third party advertisers, canprovide additional content for display on the web pages together withthe information provided by the web publishers. A content selectionserver may select certain additional content to display on a renderingof a web page based on various factors including, e.g., contentselection criteria associated with the content to be displayed. Thus, aperson viewing a web page can access the information that is the subjectof the web page, as well as selected third party content that may appearwith the web page.

SUMMARY

At least one aspect is directed to a method of selecting content fordisplay on a user device via a computer network. The method can includea data processing system receiving a search query provided via a userdevice. The data processing system can include one or more processors.The method can include a query reference module of the data processingsystem identifying, via a data structure having information aboutentities, an entity of the search query and a corresponding confidencescore. The method can include identifying a property of the entity ofthe search query. The method can include the data processing systemdetermining a match between a property of an entity of content selectioncriteria and the property of the entity of the search query. The methodcan include the data processing system selecting the content item as acandidate for display on the user device based on the match and theconfidence score.

In some implementations, the property of the entity of the search queryincludes a second entity and a relation between the entity and thesecond entity.

In some implementations, the property of the entity of the search queryincludes a query graph and the property of the entity of the contentselection criteria includes a content selection criteria graph. In someimplementations, the method includes the data processing systemcomparing, on a node-by-node basis, the content selection criteria graphwith the query graph to determine the match.

In some implementations, the method includes the data processing systemdetermining the match based on a query graph of the entity of the searchquery and a content selection criteria graph of the entity of thecontent selection criteria.

In some implementations, the method includes the data processing systemtranslating properties of the entity of the search query into a flatdata structure. The flat data structure can include informationassociated with the properties. The method can include the dataprocessing system identifying, using the flat data structure, aplurality of content selection criteria associated with the flat datastructure. In some implementations, the method includes the dataprocessing system comparing each of the plurality of content selectioncriteria with the properties of the entity of the query to identifymatching content selection criteria.

In some implementations, the method includes the data processing systemmapping the property of the content selection criteria onto the propertyof the entity of the query. In some implementations, the confidencescore indicates a semantic relevancy of the entity to the search query.In some implementations, the method includes the data processing systemdetermining that the confidence score exceeds the threshold.

At least one aspect is directed to a method of selecting content fordisplay on a user device via a computer network. In someimplementations, the method includes a data processing system having oneor more processors receiving a search query provided via a user device.The method can include the data processing system (e.g., via a queryreference module) identifying, via a data structure having informationabout entities, an entity of the search query and a correspondingconfidence score. The data processing system can generate a query graphhaving linked nodes. A node of the query graph can include the entityand a property of the entity. The method can include the data processingsystem retrieving a content selection criteria graph for a content itemof a content provider. The content selection criteria graph can includea linked node. The method can include the data processing systemdetermining a match between the content selection criteria graph and thequery graph. The method can include the data processing system selectingthe content item as a candidate for display on the user device. The dataprocessing system can select the content item as a candidate for displayon the user device based on the match and the confidence score.

In some implementations, the method includes the data processing systemgenerating the query graph using a commercially relevant subset of thedata structure having information about entities. In someimplementations, the method includes the data processing systemselecting the content item as a candidate for display responsive to thematch and the confidence score satisfying a threshold.

In some implementations, the method includes the data processing systemcomparing the content selection criteria graph with the query graph. Themethod may include comparing the content selection criteria graph withthe query graph on a node-by-node basis. In some implementations, themethod includes matching a topology of the content selection criteriagraph with the query graph.

In some implementations, the method includes translating the query graphinto a flat data structure. The flat data structure can includeinformation associated with the query graph. In some implementations,the method includes a search module identifying multiple contentselection criteria graphs associated with the flat data structure. Thesearch module may use the flat data structure to make thisidentification. In some implementations, the method includes comparingeach of the of the content selection criteria graphs with the querygraph to identify a matching content selection criteria graph. In someimplementations, the method includes determining that the contentselection graph fits within the query graph.

In some implementations, the confidence score indicates a semanticrelevancy of the entity to the search query. In some implementations,the method includes determining that the confidence score exceeds thethreshold. In some implementations, the method includes the dataprocessing system identifying multiple interpretations of the searchquery. The multiple interpretations can each include at least oneentity. Each entity can include a corresponding confidence score. Themethod may include the data processing system filtering the multipleinterpretations based on the corresponding confidence score of the atleast one entity of the multiple interpretations.

At least one aspect is directed to a system for selecting content fordisplay on a user device via a computer network. In someimplementations, the system includes a data processing system having oneor more processors. The data processing system can include an interfacemodule configured to receive a search query provided via a user device.The system can include a query reference module configured to identify,via a data structure having information about entities, an entity of thesearch query and a corresponding confidence score. The system can alsobe configured to identify a property of the entity of the search query.The system can include a matching module configured to determine a matchbetween a property of an entity of content selection criteria and theproperty of the entity of the search query. The system can include acontent selector configured to select the content item as a candidatefor display on the user device based on the match and the confidencescore.

At least one aspect is directed to a system for selecting content fordisplay on a user device via a computer network. The system can includea data processing system having one or more processors. In someimplementations, the system includes an interface module configured toreceive a search query provided via a user device. In someimplementations, the system includes a query reference module configuredto identify, via a data structure having information about entities, anentity of the search query and a corresponding confidence score. Thequery reference module can be further configured to generate a querygraph comprising linked nodes. A node of the query graph can include theentity. The data processing system can be further configured to retrievea content selection criteria graph for a content item of a contentprovider. The content selection criteria graph can include a linkednode. The system can include a matching module configured to determine amatch between the content selection criteria graph and the query graph.The system can include a content selector configured to select thecontent item as a candidate for display on the user device responsive tothe match and the confidence score satisfying a threshold.

In some implementations, the property of the entity of the search queryincludes a second entity and a relation between the entity and thesecond entity. In some implementations, the property of the entity ofthe search query comprises a query graph.

At least one aspect is directed to a computer-readable storage devicehaving processor executable instructions to select content via acomputer network. The instructions can include instructions to receive asearch query provided via a user device. The instructions can includeinstructions to identify an entity of the search query and acorresponding confidence score. The instructions can includeinstructions to identify a property of the entity of the search query.The instructions can include instructions to determine a match between aproperty of an entity of content selection criteria and the property ofthe entity of the search query. The instructions can includeinstructions to select the content item as a candidate for display onthe user device based on the match and the confidence score.

At least one aspect is directed to a non-transitory computer-readablemedium comprising processor executable instructions to select contentvia a computer network. In some implementations, the instructionsinclude instructions to receive a search query provided via a userdevice. In some implementations, the instructions include instructionsto identify an entity of the search query and a corresponding confidencescore. The instructions can include instructions to access a datastructure having information about entities to generate a query graphwith linked nodes. A node of the query graph can include the entity. Theinstructions can include instructions to retrieve a content selectioncriteria graph for a content item of a content provider. The contentselection criteria graph can include a linked node. The instructions caninclude instructions to determine a match between the content selectioncriteria graph and the query graph. The instructions can includeinstructions to select the content item as a candidate for display onthe user device responsive to the match and the confidence scoresatisfying a threshold.

At least one aspect is directed to a method of selecting content fordisplay on a user device. The method can include a data processingsystem having one or more processors receiving a query to generatecontent selection criteria. The method can include one or moreprocessors receiving an indication to generate content selectioncriteria based on target content. The method can include the one or moreprocessors identifying an entity of the target content and a property ofthe entity. The method can include the one or more processors accessing,in a database, a template having a topology and a named variablecorresponding to the property of the entity. The method can include theone or more processors determining, based on the named variable and thetopology of the template, semantic criteria matching the property of theentity. The method can include the one or more processors selectingcandidate content selection criteria based on a statistical metric ofeach of the matching semantic criteria.

In some implementations, the method includes identifying a plurality ofentities of the query and a corresponding confidence score for each ofthe plurality of entities. The method may include determining that atleast one of the plurality of entities satisfy a threshold based on thecorresponding confidence score. The method can include selecting theproperty associated with the at least one of the plurality of entities.

In some implementations, the template includes a first named variableand a second named variable. The method may include identifying a firstplurality of semantic criteria for the first named variable and a secondplurality of semantic criteria for the second named variable. The methodmay include determining a Cartesian product based on the first andsecond plurality of semantic criteria.

In some implementations, the target content includes multiple queries.The method may include determining, for the semantic criteria, a termfrequency based on the plurality of queries. The method may includedetermining, for the semantic criteria, an inverse query frequency.

In some implementations, the method may include the data processingsystem providing the candidate criteria for display to a contentprovider. The method may include the data processing system receiving aselection of the candidate criteria. The method may include associatingthe selected candidate criteria with the content group.

In some implementations, the method may include receiving target contentthat includes a plurality of queries. Content selection criteria can begenerated for a content group based on the plurality of queries. In someimplementations, the method may include receiving an indication of anonline document of the content provider. The method may include usingthe online document to generate content selection criteria for a contentgroup. In some implementations, the method may include receiving targetcontent that includes queries based on historical traffic directed to anonline document of a content provider. The method may include using thehistorical queries to generate candidate content selection criteria.

In some implementations, the method may include ranking the matchingsemantic criteria based on the statistical metric. The method mayinclude selecting the candidate content selection criteria based on therank. In some implementations, the topology includes a propertyassociated with the named variable.

At least one aspect is directed to a method of selecting content fordisplay on a user device. The method can include a data processingsystem having one or more processors receiving a query to generatecontent selection criteria. The method can include a query referencemodule of the data processing system identifying an entity of the queryand a query graph based on the entity. The method can include the dataprocessing system accessing a database to identify a templatecorresponding to the query graph. The template can include a topologyand a named variable. The method can include the data processing systemdetermining, based on the named variable and the topology of thetemplate, a plurality of semantic criteria matching the query graph. Themethod can include the data processing system using a statistical metricof each of the matching semantic criteria to select candidate contentselection criteria.

At least one aspect is directed to a system for selecting content fordisplay on a user device via a computer network. The system can includea data processing system having one or more processors. In someimplementations, the system includes an interface module configured toreceive a query to generate content selection criteria. The system caninclude a query reference module configured to identify an entity of thequery to generate a query graph. The system can include a lookup moduleconfigured to access a database to identify a template having a topologyand a named variable corresponding to the query graph. The system can befurther configured to determine, based on the named variable and thetopology of the template, a plurality of semantic criteria matching thequery graph. The system can include a matching module configured to usea statistical metric of each of the matching semantic criteria to selectcandidate content selection criteria.

At least one aspect is directed to a system for selecting content fordisplay on a user device via a computer network. The system can includea data processing system having one or more processors. In someimplementations, the system includes an interface module configured toreceive an indication to generate content selection criteria based ontarget content. The system can include a query reference moduleconfigured to identify an entity of the target content and a property ofthe entity. The system can include a lookup module configured to accessa database to identify a template having a topology and a named variablecorresponding to the property of the entity. The system can be furtherconfigured to determine, based on the named variable and the topology ofthe template, a plurality of semantic criteria matching the property ofthe entity. The system can include a matching module configured to use astatistical metric of each of the matching semantic criteria to selectcandidate content selection criteria.

At least one aspect is directed to a non-transitory computer-readablemedium comprising processor executable instructions to select contentvia a computer network. In some implementations, the instructionsinclude instructions to receive an indication to generate contentselection criteria based on target content. The instructions can includeinstructions to identify an entity of the target content and a propertyof the entity. The instructions can include instructions to access, in adatabase, a template. The template can include a topology and a namedvariable. The instructions can include instructions to determine, basedon the named variable and the topology of the template, a plurality ofsemantic criteria matching the property of the entity. The instructionscan include instructions to use a statistical metric of each of thematching semantic criteria to select candidate content selectioncriteria.

BRIEF DESCRIPTION OF THE DRAWINGS

The details of one or more implementations of the subject matterdescribed in this specification are set forth in the accompanyingdrawings and the description below. Other features, aspects, andadvantages of the subject matter will become apparent from thedescription, the drawings, and the claims.

FIG. 1 is an illustration of one implementation of a system forselecting content via a computer network.

FIG. 2 is an illustration of one implementation of selecting contentusing entity properties.

FIG. 3 is an illustration of one implementation of selecting contentusing entity properties.

FIG. 4 is an illustration of one implementation of a method of selectingcontent via a computer network.

FIG. 5 is an illustration of one implementation of a system forselecting content via a computer network.

FIG. 6 is an illustration of one implementation of creating contentselection criteria using entity properties.

FIG. 7 is an illustration of one implementation of a method of selectingcontent via a computer network.

FIG. 8 is a block diagram illustrating a general architecture for acomputer system that may be employed to implement various elements ofthe systems and methods described herein, in accordance with animplementation.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

Systems and methods of this disclosure are directed generally towardsselecting online content for display alongside search query results. Thesystems and methods can generate or use a form of selection criteriathat is based on properties of entities mentioned in queries, ratherthan based on keywords and synonyms of keywords mentioned in queries. Insome implementations, the systems and methods use information from acontent provider to create content selection criteria, receive a userquery, generate a dynamic set of entities corresponding to the contentselection criteria and a dynamic set of entities corresponding to thesearch query, and determine whether the set of entities of the searchquery matches the set of entities of the content selection criteria.

In an illustrative implementation, a content provider may provide thefollowing content selection criteria “all queries that mention books byPerson X”. The system may first identify an entity of the criteria(e.g., Person X). The entity may correspond to a unique entityidentifier. The system may then access a data structure providingstructured and detailed information about persons, places or thingsassociated with unique entity identifiers. The system may use the datastructure to identify properties of the entity ID, such as a secondentity and its relation to the entity. The entity “Person X” may berelated to a book “Title Y” and the relation may be “author of”. Theentity “Person X” may also be related to a movie “Title Z” and therelation may be “producer of”. The system may identify, use or obtainthe properties, generate a subset of the data structure based on thecontent providers content selection criteria.

Thereafter, the system may receive a search query provider via a userdevice. The system can identify one or more entities of the searchquery. The system can further identify properties of the entity of thesearch query using the data structure having entity information. Thesystem may annotate or otherwise associate this information with one ormore entities of the search query. In an illustrative implementation,the data structure having entity information may include an entitygraph, and the system may generate a replica of a subset of the entitygraph where the search query of the user is inserted as a node in thereplica, thus creating a search query graph.

In another illustrative implementation, if the search query provided bythe user was “Album X”, the system can determine at serving time thatthe query is about a specific entity, that the entity is an album by“Singer Y”, that the album has a song titled “Song Z”, the album wasproduced in 1983, etc. Thus, all of these properties are eligible foruse as content selection criteria. The system can identify the followingcontent selection criteria as a match: “all queries mentioning Albums bySinger Y”.

In some implementations, the system receives a search query input into asearch engine via a user device. Using the received user search query,the system generates a replica of a subset of an entity graphcorresponding to the entities mentioned in the search query, where thesearch query is inserted as a node in the replica. The technology thenmatches this replica entity graph with other entity graphs correspondingto a content provider's content selection criteria. If the entity graphcorresponding to the content provider's content selection criteria mapsonto the search query entity graph or otherwise matches the replicaentity graph, then it is a match. The system may then select contentitems of the content provider corresponding to the matching entitygraph.

In some implementations, the systems and methods create contentselection criteria used to select online advertisements for display on auser device alongside search engine results. Target content provided bya content provider, such as a set of illustrative queries (or a landingpage) can be used to generate semantic criteria based on extractedsemantic features of the target content, queries or landing pageprovided by the content provider.

In some implementations, target content or illustrative queries (or alanding page) is received from the advertiser. In an illustrativeimplementations, the illustrative queries may include a list of books byan author. Semantic features of the illustrative queries can beextracted to determine whether these extracted semantic features aresignificantly more common than random chance (e.g., rank odds ratio ofeach matching criteria). In some implementations, the system analyzesthe list of books to identify the following matching criteria: the bookswere all authored by “Author X”, take place in the same fictionaluniverse, are all fantasy books, etc. The system can prompt theadvertiser with these extracted semantic features and their associatedentities. The system can also show the advertiser what user searchqueries would trigger a match, and provide an estimate of trafficvolume.

FIG. 1 illustrates one implementation of a system 100 for selectingcontent via a computer network such as network 105. The system 100 andits components, such as a data processing system 120, may includehardware elements, such as one or more processors, logic devices, orcircuits. The network 105 can include computer networks such as theInternet, local, wide, metro, data, or other area networks, intranets,satellite networks, combinations thereof, and other communicationnetworks such as voice or data mobile telephone networks. The network105 can be used to access information resources such as web pages, websites, domain names, or uniform resource locators that can be displayedon at least one user device 110, such as a laptop, desktop, tablet,personal digital assistant, smart phone, mobile computing devices,mobile telecommunication device, wearable computing device, or portablecomputer. In one implementation, via the network 105 a user of the userdevice 110 can access web pages provided by at least one contentpublisher 115 (e.g., a web site operator). In this implementation, a webbrowser of the user device 110 can access a web server of the contentpublisher 115 to retrieve a web page for display on a monitor of theuser device 110. The content publisher 115 generally includes an entitythat operates the web page. In one implementation, the content publisher115 includes at least one web page server that communicates with thenetwork 105 to make the web page available to the user device 110.

Although FIG. 1 shows a network 105 between the user device(s) 110, dataprocessing system 120, content provider 125, and content publisher 115,the user device(s) 110, content publisher 115, content provider 125 anddata processing system 125 may be on the same network 105. The network105 can be a local-area network (LAN), such as a company Intranet, ametropolitan area network (MAN), or a wide area network (WAN), such asthe Internet or the World Wide Web. In some implementations, there aremultiple networks 105 between the user devices 110 and the dataprocessing system 120, content provider 125, and content publisher 115.In one of these implementations, the network 105 may be a publicnetwork, a private network, or may include combinations of public andprivate networks.

The network 105 may be any type or form of network and may include anyof the following: a point-to-point network, a broadcast network, a widearea network, a local area network, a telecommunications network, a datacommunication network, a computer network, an ATM (Asynchronous TransferMode) network, a SONET (Synchronous Optical Network) network, a SDH(Synchronous Digital Hierarchy) network, a wireless network and awireline network. In some implementations, the network 105 may include awireless link, such as an infrared channel or satellite band. Thetopology of the network 105 may include a bus, star, or ring networktopology. The network may include mobile telephone networks using anyprotocol or protocols used to communicate among mobile devices,including advanced mobile phone protocol (“AMPS”), time divisionmultiple access (“TDMA”), code-division multiple access (“CDMA”), globalsystem for mobile communication (“GSM”), general packet radio services(“GPRS”) or universal mobile telecommunications system (“UMTS”). In someimplementations, different types of data may be transmitted viadifferent protocols. In other implementations, the same types of datamay be transmitted via different protocols.

The system 100 can include at least one data processing system 120. Thedata processing system 120 can include at least one logic device such asa computing device having a processor to communicate via the network 105with the user device 110, the content publisher 115, and at least onecontent provider 125. The data processing system 120 can include atleast one server. In one implementation, the data processing system 120can include a plurality of servers located in at least one data center.In some implementations, the data processing system 120 may includemultiple, logically-grouped servers and facilitate distributed computingtechniques. In one of these implementations, the logical group ofservers may be referred to as a server farm or a machine farm. Inanother of these implementations, the servers may be geographicallydispersed. In other implementations, a machine farm may be administeredas a single entity. In still other implementations, the machine farmincludes a plurality of machine farms. The servers within each machinefarm can be heterogeneous—one or more of the servers or machines canoperate according to one type of operating system platform.

In one implementation, servers in the machine farm may be stored inhigh-density rack systems, along with associated storage systems, andlocated in an enterprise data center. In this implementation,consolidating the servers in this way may improve system manageability,data security, the physical security of the system, and systemperformance by locating servers and high performance storage systems onlocalized high performance networks. Centralizing the servers andstorage systems and coupling them with advanced system management toolsallows more efficient use of server resources.

Management of the machine farm may be de-centralized. In oneimplementation, one or more servers may comprise components, subsystemsand circuits to support one or more management services for the machinefarm. In one of these implementations, one or more servers providefunctionality for management of dynamic data, including techniques forhandling failover, data replication, and increasing the robustness ofthe machine farm. Each server may communicate with a persistent storeand, in some implementations, with a dynamic store.

Server may include a file server, application server, web server, proxyserver, appliance, network appliance, gateway, gateway, gateway server,virtualization server, deployment server, secure sockets layer virtualprivate network (“SSL VPN”) server, or firewall. In one implementation,the server may be referred to as a remote machine or a node.

The data processing system 120, content provider 125, content publisher115, and user device 110 may be deployed or executed on any type ofclient or server, or any type and form of computing device, such as acomputer, network device or appliance capable of communicating on anytype and form of network and performing the operations described herein.

In one implementation, the data processing system 120 includes a contentplacement system having at least one server. The data processing system120 can also include at least one interface module 135, at least onequery reference module 140, at least one matching module 145, at leastcontent selector 150, and at least one database 155. In someimplementations, the data processing system includes a search module.The at least one interface module 135, at least one query referencemodule 140, at least one matching module 145, at least content selector150, and at least one search module can each include at least oneprocessing unit or other logic device such as programmable logic arrayengine, or module configured to communicate with the database 155. Theat least one interface module 135, at least one query reference module140, at least one matching module 145, at least content selector 150,and at least one search module can be separate components, a singlecomponent, or part of the data processing system 120.

In some implementations, the data processing system 120 obtainsanonymous computer network activity information associated with aplurality of user devices 110. A user of a user device 110 canaffirmatively authorize the data processing system 120 to obtain networkactivity information corresponding to the user's user device 110. In oneimplementation, the data processing system 120 can prompt the user ofthe user device 110 for consent to obtain one or more types of networkactivity information, such as geographic location information. Theidentity of the user of the user device 110 can remain anonymous and theuser device 110 may be associated with a unique identifier (e.g., acookie).

For situations in which the systems discussed here collect personalinformation about users, or may make use of personal information, theusers may be provided with an opportunity to control whether programs orfeatures that may collect personal information (e.g., information abouta user's social network, social actions or activities, a user'spreferences, or a user's current location), or to control whether or howto receive content from the content server that may be more relevant tothe user. In addition, certain data may be treated in one or more waysbefore it is stored or used, so that certain information about the useris removed when generating parameters (e.g., demographic parameters). Inone implementation, a user's identity may be treated so that noidentifying information can be determined for the user, or a user'sgeographic location may be generalized where location information isobtained (such as to a city, ZIP code, or state level), so that aparticular location of a user cannot be determined. Thus, the user mayhave control over how information is collected about the user and usedby a content server.

In one implementation, the data processing system 120 receives contentor content items from a content provider 125, such as a commercialentity, online retailer, business, advertiser, individual or any entitythat wants to provide content for display on a user device 110 via thecomputer network 105. The content or content items may include, e.g.,text, characters, symbols, images, video, audio, or multimedia content.In one implementation, a content item may include an onlineadvertisement, article, promotion, coupon, or product description. Inaddition to receiving content from a content provider 125, the dataprocessing system 120 may receive location information (e.g., aredemption location, retail store, restaurant location, point of salelocation, etc.) associated with the content provider 125 that providesthe content, or the commercial entity associated with the providedcontent in the event a third-party is providing the content to the dataprocessing system 120 on behalf of a commercial entity (e.g., anadvertiser creating and providing advertisements for a retail store).The data processing system 120 can store, in database 150, the locationas a location extension. Since a content provider 125 may have multiplecontent campaigns (e.g., advertisement campaigns that include multipleadvertisements for the same or similar landing page), in oneimplementation, the location extension can be stored in a contentselection data structure associated with the content provider 125 ratherthan each individual content, content campaign or content group (e.g.,multiple content having similar keywords or content selection criteria).In one implementation, the location extension can be associated with acontent provider's 125 unique identifier when a content provider 125establishes or sets up a content campaign or provides content to thedata processing system 120.

In some implementations, the data processing system 120 includes acontent selector 150 designed and constructed to select a content itembased on a search query input via user device 110. The data processingsystem 120 may parse, analyze, match, or otherwise process one or moresearch terms of the search query to identify one or more candidatecontent items associated with the search query. In an illustrativeimplementations, the data processing system 120 may receive a searchquery comprising the term “pizza”. The data processing system 120 maythen parse a data structure to identify content items related to pizza,such as advertisements or coupons for pizza restaurants. These contentitems may be provided by one or more content providers 125. In someimplementations, the data processing system 120 may select one or morecontent items to provide for display on the user device based on, e.g.,an online auction, advertisement score, keyword score, location, orother criteria. When the content item is presented to a user via theuser device 110, the data processing system may receive an indication ofinterest in the content item (e.g., a click, selection, etc.). In someimplementations, the data processing system 120, responsive to receivingan indication of user interest in the content item, may bill or chargeor otherwise request consideration from the content provider 125associated with the content item.

The data processing system 120 may provide the content item to the webpage for display in response to receiving a request for content from acomputing device such as, e.g., user device 110. In someimplementations, the data processing system 120 receives the request viaan application executing on the user device 110. In someimplementations, a mobile application executing on a mobile device(e.g., smart phone or tablet) may make a request for content. In someimplementations, a web page may request content from the data processingsystem 120 responsive to a user of a user device 110 visiting the webpage. In some implementations, the data processing system 120 mayreceive a request for content via a search engine and responsive to auser of a user device 110 entering a search query.

In some implementations, the request for content includes informationthat can facilitate content selection. In some implementations, the dataprocessing system 120 may request information from the user device 110to facilitate identifying content or selecting content. The dataprocessing system 120 may request or obtain information responsive toreceiving a request for content from the user device 110. Theinformation may include information about displaying the content on theuser device 110 (e.g., a content slot size or position) or availableresources of user device 110 to display or otherwise manipulate thecontent.

Responsive to a request for content from a web page operator 115, thedata processing system 120 provides a content item for display with aweb page on a user device 110. A user of the user device 110 may viewthe content item (e.g., an impression) or may click on or select thecontent item (e.g., a click). In one implementation, an indication ofuser interest in the content item may include a click, selection, mouseover, finger gesture, shake motion, voice command, tap, or anotherindication that indicates user interest in the content item. In someimplementations, the indication of user interaction may include the userusing the content item (e.g., a coupon) to make a purchase at aredemption location.

In one implementation, the data processing system 120 includes aninterface module 135 designed and constructed to receive, access,obtain, transmit, convey or otherwise communicate with one or morecomponent of the data processing system 120 or device (e.g., contentprovider 125, content publisher 115 and user device 110) via network105. In some implementations, the interface module 135 is configured toreceive a search query provided via a user device 110. The search querymay be input into a search engine of, associated with, or otherwisecommunicatively coupled to data processing system 120. In someimplementations, the data processing system 120 may store the user'ssearch query in a database 155 for later processing. In someimplementations, the data processing system 120 provides or otherwiseconveys the user search query to the query reference module 140 forfurther processing. In some implementations, the interface module 135receives content selection criteria information from a content provider125 and stores this information in a database 155 or otherwise transmitsor conveys the information to one or more component of the dataprocessing system 120 for further processing.

In one implementation, the data processing system 120 includes a queryreference module 140 designed and constructed to identify entities in asearch query. An entity may be a single person, place or thing, and therepository can include millions of entities that each have a uniqueidentifier to distinguish among multiple entities with similar names(e.g., a Jaguar car versus a jaguar animal). The data processing systemcan access a reference entity and scan arbitrary pieces of text (e.g.,text in web pages, text of keywords, text of content, text ofadvertisements) to identify entities from various sources. One suchsource may be a manually created taxonomy of entities such as an entitygraph of people, places, properties, and things, built by a community ofusers.

In some implementations, the data processing system 120 obtains aclassification of a plurality of entities. An entity may be a singleperson, place, thing or topic. Each entity has a unique identifier thatmay distinguish among multiple entities with similar names (e.g., aJaguar car versus a jaguar animal). A unique identifier (“ID”) may be acombination of characters, text, numbers, or symbols. The dataprocessing system may obtain the classification from an internal orthird-party database via network 105. In one implementation, theentities may be manually classified by users of a user device 110. Insome implementations, users may access the database of entities vianetwork 105. Users may upload at least one entity or upload multipleentities in a bulk upload. Users may classify the uploaded entities, orthe upload may include the classification of at least one entity. Insome implementations, upon receiving an entity, the data processingsystem 120 may prompt the user for a classification.

In some implementations, entities may be manually classified by users.Classifications may indicate the manner in which entities arecategorized or structured, e.g., ontology. In some implementations, anontological classification may include attributes, aspects, properties,features, characteristics, or parameters that entities can have.Ontological classifications may also include classes, sets, collections,concepts, or types. An ontology of “vehicle” may include: type—groundvehicle, ship, air craft; function—to carry persons, to carry freights;attribute—power, size; component—engine, body; etc. In someimplementations, the manual classification includes structured data thatprovides a manually created taxonomy of entities. In someimplementations, entities may be associated with an entity type, such aspeople, places, books, or films. In some implementations, entity typesmay include additional properties, such as date of birth for a person orlatitude and longitude for a location. Entities may also be associatedwith domains, such as a collection of types that share a namespace,which includes a directory of uniquely named objects (e.g., domain nameson the internet, paths in a uniform resource locator, or directors in acomputer file system). Entities may also include metadata that describesproperties (or paths formed through the use of multiple properties) interms of general relationships.

The data processing system 120 or a user of user device 110 may classifyan entity based on a domain, type, and property. In someimplementations, a domain may be American football and have an ID“/american_football”. This domain may be associated with a head coachtype with ID “/American_football/football_coach”. This type may includea property for current team head coached with ID“/American_football/football_coach/current_team_head_coached”. Eachdomain, type, property or other category may include a description. Inan illustrative implementations, “/American_football/football_coach” mayinclude the following description: “‘Football Coach’ refers to coachesof the American sport Football.” In some implementations, the dataprocessing system 120 can scan text or other data of a document andautomatically determine a classification. The data processing system 120may scan information resources via network 105 for information aboutfootball coaches, and classify that information as“/American_football/football_coach”. The data processing system 120 mayfurther assign the entity football coach a unique identifier thatindicates a classification.

Entities may be classified, at least in part, by one or more humans(“entity contributors”). This may be referred to as manualclassification. In some implementations, entities may be classifiedusing crowd sourcing processes. Crowd sourcing may occur online oroffline and may refer to a process that involves outsourcing tasks to adefined group of people, distributed group of people, or undefined groupof people. Users may add, modify, or delete classifications online. Anillustrative implementation of offline crowd sourcing may includeassigning the task of uploading or classifying entities to an undefinedpublic not using the network 105, e.g., to students in a classroom orpassersby on the street or at a mall.

In some implementations, data processing system 120 may obtain or gainaccess to the classification of a plurality entities from contentrepository 155 (e.g., a database) or another database accessible vianetwork 105. In some implementations, entities may be stored in a graphdatabase where the entity data structure includes as a set of nodes anda set of links that establish relationships between the nodes. Theentity data structure in the graph database may be non-hierarchical,which may facilitate modeling complex relationships between individualelements, and allow entity contributors to enter new objects andrelationships into the underlying graph structure.

In some implementations, the data processing system 120 identifies anentity of a search query provided by a user device 110 (e.g., input intoa search engine). The data processing system 120 includes a queryreference module 140 that determines an entity of the search query. Thequery reference module may identify zero, one or many entities in orassociated with the search query. The data processing system may mapterms, keywords, or phrases in the search query to one or more welldefined entities in a database. The data processing system 120 may scorethe entities based on the relations among entities in the database andselect the entities with the highest weight as page entities. The dataprocessing system 120 may further assign a confidence score to theentity, and select, for further processing, the entity with the highestconfidence score. The confidence score may reflect the likelihood thatthe identified entity in the database semantically matches the searchquery.

In some implementations, the query reference module 140 may identifymultiple interpretations of the search query, where each interpretationincludes one or more entities and an individual confidence score withinthat interpretation. In an illustrative implementation, a search query“flight from springfield to paris” may have multiple interpretationsbecause there are multiple cities or towns named “Springfield”. Thus,each interpretation of “Springfield” might have its own unique entityidentifier for a specific “Springfield” instance, while the uniqueentity identifier for “Paris” may remain the same. In someimplementations, upon identifying multiple interpretations of the searchquery, where at least two of the interpretations include at least oneentity, the data processing system 120 may filter, select or otherwiseidentify interpretations or entities to use based on the confidencescore of the entity. The data processing system may identify entities ofinterpretations that exceed a threshold. In some implementations, thedata processing system 120 may determine an average confidence score foran interpretation, a weighted average of confidence score, or otherwisedetermine a confidence score, accuracy or quality for the overallinterpretation in order to select an interpretation for contentselection.

The identified entities can include additional information about theclassification (e.g., metadata). In some implementations, the additionalinformation may include a domain, type, property, or description. Insome implementation, the entity includes a unique identifier thatindicates a classification of the entity. The additional information maybe inferred via the unique identifier of the entity. In an illustrativeimplementation, an entity may be French, with a unique identifier“/dining/cuisine”. The unique identifier “/dining/cuisine” may includeproperties such as description, region of origin, restaurants,ingredients, dishes, or chefs.

The data processing system 120 may obtain some or all of the additionalinformation associated with the entity and annotate, decorate orotherwise associate that information with the entity. The additionalinformation may be linked to the entity, where the link includes arelationship. In an illustrative implementation, if the entity is anfilm, the data processing system 120 can annotate the entity with thefollowing properties: produced by, business/product_line/category,written by, genre, featured film locations, production companies, filmcountry, etc.

In some implementations, the data processing system 120 only annotatesthe entity with commercially relevant information. In an illustrativeimplementation, the data processing system 120 may access a datastructure having a subset of the entire entity graph that ispredetermined to be commercially relevant. The predetermined subset maybe generated or uploaded by one or more of an administrator of the dataprocessing system 120, content provider 125, user device 110, via crowdsourcing techniques, etc. In some implementations, commercially relevantinformation may include information for which a content provider maywant to provide content items (e.g., advertisements). In someimplementations, commercial relevant may refer to content selectiongoals of the content provider 125. In an illustrative implementation, anentity property /film/film/directedby may be commercially relevant (orvaluable) to a content provider 125 whose goal is to select contentitems based on films by specific directors.

In one implementation, the data processing system 120 includes amatching module 145 designed and constructed to identify and determinewhether the search query satisfies content selection criteria providedby a content provider in order to select content item of the contentprovider. The matching module 145 can retrieve, receive, obtain orotherwise identify one or more content selection criteria graphs for acontent item, content group, or content campaign of one or more contentproviders. The content selection criteria graph includes a linked node.In some implementations, the content selection criteria graphs refers toidentifying an entity and properties associated with that entity (e.g.,an entity X is related to entity Y via the relation Z). In anillustrative implementation, as shown in FIG. 2, the content selectioncriteria graph 226 may include a content selection query provided by acontent provider, which is linked to an entity, which is linked to aproperty, which has a value. In the search query graph 200 depicted inFIG. 2, an entity may include “The Film” and a property of the entitymay include “produced_by” entity “Person_A”. The entity “The Film” isrelated to entity “Person_A” via relationship “produced_by”. In someimplementations, a property of the entity includes a second entity and arelation between the entity and the second entity.

The data processing system 120 identifies a matching content selectioncriteria graph (e.g., properties associated with an entity of a contentselection criteria). In some implementations, to identify the matchinggraph or properties, the data processing system 120 (e.g., via a searchmodule), may further process the search query graph or propertiesgenerated or identified by the query reference module 140 to create flatdata structure (or list) of all the information in the search querygraph. In some implementations, the data processing system 120 maytranslate the search query graph into the flat data structure or listform, which includes some or all of the facts in the search query. In anillustrative implementation, a search query may include “New York City”,and the flat data structure or list may include the followinginformation or entries: “query mentions New York City”; “query mentionsa place within New York Sate”; “query mentions a city”; “query mentionsa place in United States”, etc. As this is a flat data structure orlist, it may not include all of the relationships or links included inthe search query graph. In some implementations, the data processingsystem can employ distributed computing on clusters of computers usinglibraries or programming for processing large data sets.

Using this list, the data processing system 120 (e.g., via a searchmodule) can parse, search or otherwise access a database having contentselection criteria provided by multiple content providers to identifymatching content selection criteria. The data processing system 120 canuse various techniques to identify the content selection criteria. Insome implementations, the data processing system 120 uses an orderedtree data structure that stores a dynamic set or associated array, suchas a tie, radix tree, prefix tree, etc. In some implementations, thesearch module includes an API configured to register the list of facts(e.g., a query format) to match against documents having the contentselection criteria. For every search criteria document presented, thesearch module can return an identifier for the search criteria of all ofthe matching queries. In some implementations, the search module mayhandle a potentially large number of matched queries using queue.

The data processing system 120 may identify multiple content selectioncriteria that could potentially match the search query graph. In someimplementations, the search query graph refers to properties of anentity of a search query. Since the data processing system 120 retrievesthe potentially matching content selection criteria using the flat datastructure or list, which is the translated search query graph, the dataprocessing system 120 may then determine, one a node-by-node basis,whether the retrieved content selection criteria matches the user searchquery. That is, because the flat data structure may not include therelationship links between each listed fact (e.g., which is conveyed viathe topology of the search query graph), the data processing system 120(e.g., via the matching module 145), may further compare the retrievedcontent selection criteria graphs with the search query graph todetermine whether the search query satisfy a content selection criteriaof a content provider. In some implementations, the matching module 145may compare the structure, topology, properties, predicates,relationship, links or other aspects of the search query graph and thecontent selection criteria graph to identify a match.

In some implementations, the data processing system 120 matches thecontent selection criteria graph with the search query graph by mappingthe content selection criteria graph onto the search query graph. Insome implementations, the data processing system 120 matches a propertyof an entity of a content selection criteria with a property of anentity of a search query. In some implementations, a property of theentity includes a second entity and a relation between the entity andthe second entity. The data processing system 120 may determine there isa one-to-one correspondence between the content selection criteria graphand the search query graph, or that the content selection criteria graphotherwise fits onto, maps to, or matches the search query graph.

In some implementations, the data processing system 120 employs astep-through process to identify whether the content selection criteriamatches the user search query. The data processing system 120 may stepthrough each node and link in the content selection criteria graph todetermine whether the search query graph includes the corresponding nodeor link. If the search query graph does not include the correspondingnode or link, the data processing system 120 may determine that thecontent selection criteria graph does not match, and move on to the nextcontent selection criteria graph and again step-through the links andnodes. In some implementations, the data processing system may comparemultiple content selection criteria graphs in parallel (e.g., viadistributed computing architecture). Thus, in some implementations, thedata processing system 120 matches both the content of the node or linkand the topology.

The data processing system 120 may identify zero, one or multiplematching content selection criteria graphs. In an illustrativeimplementation, the data processing system 120 may identify zeromatching content selection criteria graphs (based on topology andcontent) in the event the flat data structure returned false positives(i.e., graphs with matching content, but without matching topology orsequence).

Upon identifying a content selection criteria graph that matches thesequence of properties, links or nodes of the search query graph, thedata processing system 120 (e.g., via a content selector 150) maydetermine that content items associated with this content selectioncriteria graph are eligible to be provided to a user device 110 thatprovided, or is otherwise associated with, the search query. In someimplementations, the content selection criteria may be associated with acontent campaign (e.g., multiple content groups based on a commontheme), a content group (e.g., multiple content items associated with acommon landing page), or a content item (e.g., an advertisement, onlinedocument, etc.).

FIG. 2 is an illustration of one implementation of selecting contentusing entity properties. The illustration depicts an implementation of asearch query graph 200 and a content selection criteria graph 226. Inthis illustrative implementation, a data processing system receives asearch query “the film” 202. The search query may have been input into asearch engine via a user device. The data processing system candetermine (e.g., via a query reference module) that the search query 202mentions 204 one entity 206 (e.g., Mention0). The data processing systemfurther determines that Mention0 206 (which is an entity with a uniqueentity ID), has a confidence score of 0.85 (208). The data processingsystem identifies the entity with a unique entity identifier EntityID_1and with the string “The Film” (210). The string “The Film” is one wayof rendering the EntityID_1, but it can be rendered in various ways suchas different languages, fonts, sizes, symbols, audio, video, multimedia,etc. In some implementations, the data processing system may not renderthe entity ID in order to select content. In some implementations, thedata processing system may render a human interpretable format uponrequest or when generating a report.

In some implementations, the entity 210, relation 212, and entity 220may be referred to as a triple comprising a subject, predicate, andobject, respectively. The data processing system can further annotate,decorate, or otherwise associate the entity 210 with additionalinformation, such as the following properties: /film/film/produced_by(212) /m/EntityID_2 “Person_A” (220); /business/product_line/category(214) /m/EntityID_3 “DVDs & Videos” (222); /film/film/directed_by (216)/m/EntityID_2 “Person_A” (220), and film/film/genre (218) /m/EntityID_4“Horror” (224). These relations 212-218 may be referred to aspredicates. The properties may be represented in various ways including,e.g., via unique identifier, different languages, symbols, characters,colors, strings, different structures, etc. The properties may alsoinclude various granularity. In some implementations, the granularitymay include “directed_by” while in other implementations the granularityincludes “/film/film/directed_by”. In some implementations, theidentified relations may represent a subset of all available relationsfor an entity. In some implementations, this subset may correspond tocommercially relevant properties. Each predicate 212-218 may include, orbe linked to, one or more objects 220-224. The objects 220-224 may be anentity and identified by a unique entity identifier. In thisillustrative implementation the predicates and objects are linked asfollows: /film/film/produced_by 212 is linked to /m/EntityID_2“Person_A” 220; /business/product_line/category 214 is linked to/m/EntityID_3 “DVDs & Videos” 222; /film/film/directed_by 216 is alsolinked to /m/EntityID_2 “Person_A” 220; and film/film/genre 218 islinked to /m/EntityID_4 “Horror” 224.

In some implementations, a data processing system identifies, generates,obtains, or accesses search query graph 200 or properties of an entityof a search query. That is, the data processing system determines thatthe search query “the film” 202 mentions 204 a first entity 206 ofEntityID_1 “The Film” 210 with a confidence score of 0.85 (208), andthat the EntityID_1 (210) is related to: EntityID_2 “Person_A” (220) viarelation “/film/film/produced_by” 212; EntityID_3 “DVDs & Videos” (222)via relation “/business/product_line/category” 214; and EntityID_4“Horror” (224) via relation “/film/film/genre” 218.

In addition to the search query graph 200 based off the search query“the film” 202, the data processing system identifies one or morecandidate content selection criteria graphs 226 that may match thesearch query graph for the search query 202. The content selectioncriteria graph 226 is based off of a content selection criteria 228 thatis provided by a content provider. In this illustrative implementation,the topology is as follows: query 228, mentions 230, single mention 232,entity 234, single entity 236, predicate /film/film/directed_by 238, andobject /m/EntityID_2 “Person_A” 240.

Upon identifying this content selection criteria graph 226 as acandidate for matching, the data processing system may determine whetherthe content selection criteria graph 226 (or properties of an entity ofcontent selection criteria) matches the search query graph 200 (orproperties of an entity of a search query) for search query 202. Thedata processing system may determine that the topology and the contentmatches by performing a node-by-node or step-by-step basis comparison ofthe two graphs 200 and 226 (e.g., as shown by matching lines 250). In anillustrative implementation, the data processing system determine thatthe query 228 nodes corresponds to the search query node 202, mentions230 corresponds to mentions 204, mention0 206 corresponds to the singlemention 232, entity confidence score 208 satisfies an entity threshold234 (e.g., a predetermined entity threshold, a threshold set by thecontent provider in the content selection criteria, a dynamic thresholdadjusted based on performance feedback such as click through rate orconversion rate on content items provided for display), /m/EntityID_1“The Film” 210 corresponds to entity level 236, predicate/film/film/directed_by 238 matches film/film/directed_by 216, and/m/EntityID_2 “Person_A” 240 matches /m/EntityID_2 “Person_A” 220. Thus,the data processing system may determine that the topology and thecontent of the content selection criteria graph 226 and the search querygraph 200 match.

In some implementations, the threshold may include a quality thresholdthat is determined using based on machine learning approaches (e.g.,logistic regression) or other experimentation. The threshold mayrepresent a balance between accuracy (e.g., quality) of the entityinterpretation and the coverage (e.g., number of retrieved contentitems). In an illustrative implementation, the data processing system(e.g., via a query reference module) may identify multipleinterpretations of a search query, where each interpretation includesone or more entities and an individual confidence score within thatentity. Some entities may have a high confidence score, while otherentities may have a low confidence score. The threshold may act tofilter out the entities that are used for content selection based on theconfidence score. Thus, by lowering the threshold, more entities may beeligible for use in content selection, which may result in greatercoverage (e.g., a given content selection criteria graph may match moreuser search queries). However, using entities with a lower confidencescore may result in less relevant content selection. Thus, if thethreshold is too low, then the content items associated with the contentselection criteria may be less relevant to the search query.

If the data processing system determines that the sequence of nodes andlinks of the content selection criteria graph 226 matches that of thequery graph based on query 202, the data processing system may determinethat content items associated with the content selection criteria graph226 are eligible for selection. The data processing system may providethese content items to an auction or otherwise determine whether thesecontent items will be provided to the user device that provided thesearch query the film 202.

FIG. 3 is another illustration of one implementation of selectingcontent using entity properties. The illustration depicts a search querygraph 300 and a content selection criteria graph 340. In thisillustrative implementation, the content selection criteria 340 mentionstwo entities, as opposed to the single entity mentioned in theillustrative implementation of FIG. 2.

The data processing system may receive a search query 302 that includes“hotels near potsdamer platz”. The data processing system may determinesthat this search query has two mentions 304 and 306. Mention 304 maycorresponds to Mention1 308. Mention1 may correspond to entity/m/EntityID_7 “Potsdamer Platz” 312 and have a corresponding confidencescore 310 of 0.56. The second mention 306 corresponds to Mention0 andcorresponds to entity /m/EntityID_6 “Hotel” 328 and corresponds to aconfidence score 326 of 0.87.

The data processing system may determine that entity Potsdamer Platz 312is associated with predicates /location/location/containedby 314,/travel/tourist_attraction/near_travel_destination 316 and/location/location/containedby 318. The data processing system mayfurther identify that each of these predicates 314-318 are associatedwith objects or entities as follows: /location/location/containedby 314is linked to object /m/EntityID_8 “Europe” 320;/travel/tourist_attraction/near_travel_destination 316 is linked to/m/EntityID_5 “Berlin” 322; and /location/location/containedby 318 isalso linked to object /m/EntityID_5 “Berlin” 322. These predicates andobjects may reflect a subset of all available predicates and objects forthe entity 312. In some implementations, this subset represents acommercially relevant subset of information for this entity, or anothersubset that facilitates content selection.

The data processing system may further determine that the second entity,which is /m/EntityID_6 “Hotel” 328, is associated withpredicate/common/topic/notable_types 330, which is linked to object/m/EntityID_9 “Accommodation type” 332.

Upon generating, obtaining, or otherwise identifying the search querygraph 300 based on query 302, the data processing system may identifyone or more potential content selection criteria graphs 340 (e.g., viathe flat data structure) and determine a match based on the topology orsequence of nodes and the content on a node-by-node bases. The dataprocessing system steps through levels of the graphs 300 and 340 todetermine a match (e.g., as shown by matching lines 370). In thisillustration, the data processing system identifies content selectioncriteria graph 340 as a candidate criteria graph. The criteria providedby the content provider of graph 340 is “query (mentions entity/location/location/containedby /m/EntityID_5 “Berlin” AND mentionsentity /m/EntityID_6 “Hotel”).

In this illustrative implementation, the first level query 342corresponds to search query 302. The second level of the contentselection criteria graph 340 includes two mentions 344 and 346, whichcorresponds to mentions 306 and 304, respectively. The third levelincludes the links between the mentions and the entity, and blocks 354and 348 correspond to Mention0 324 and Mention1 308, respectively. Thefourth level includes the entities 356 and 350, which corresponds to theentity confidence scores of 0.87 (326) and 0.56 (310), respectively. Thecontent selection criteria graph 340 may include an entity thresholdvalue and the data processing system may determine a match if theconfidence score of the entity satisfies the threshold (e.g., equals orexceeds the threshold, etc.). The confidence score may represent thesemantic relevancy of the entity to the search query. The confidencescore may be based on other terms in the search query, browsing historydata, or other information that facilitates determining a semanticrelevancy of the entity to the search query.

The fifth level of the content selection criteria graph 340 includes/m/Entity ID Hotel 358, which matches the /m/EntityID Hotel 328 of thesearch query graph 300. The fifth level also includes entity block 359,which matches /mEntityID Potsdamer Platz 312. The sixth level of thecontent selection criteria graph 340 includes a predicate for the entity359, which is /location/location/containedby 360. The data processingsystem may determine that predicate 360 matches predicate 318 of thesearch query graph 300, which is also /location/location/containedby.Finally, the data processing system, may determine that the object 362/m/EntityID Berlin matches the object 322. Thus, the data processingsystem may, by comparing the content selection criteria graph 340 andsearch query graph 300 on a node-by-node basis, may determine that thetopology or sequence of the two graphs 300 and 340 match. Therefore, thedata processing system may determine that content items associated withcontent selection criteria graph 340 are eligible to be selected fordisplay or otherwise provided to the user device that provided thesearch query 302.

FIG. 4 is an illustration of one implementation of a method 400 ofselecting content via a computer network. The method 400 can beperformed via system 100 or any component thereof. In brief overview, atstage 405, the method 400 includes a data processing system receiving asearch query provided by a user device. At stage 410, the method 400includes identifying an entity of the search query and a confidencescore. At stage 415, the method 400 includes generating or identifying aquery graph with linked nodes (or otherwise identifying properties of anentity of a query). At stage 420, the method 400 includes retrieving oridentifying a content selection criteria graph for a content item (orotherwise identifying properties of an entity of a content selectioncriteria). At step 425, the method 400 includes determining a matchbetween the content selection criteria graph and the query graph. Atstep 430, the method 400 includes selecting the content item as acandidate for display on the user device.

At stage 405, the method 400 includes a data processing system receivinga search query provided by a user device. In some implementations, thedata processing system receives the search query via a network. In someimplementations, the search query may include terms, words, phrases,characters, symbols, or audio (e.g., a voice initiated search,conversational search, etc.). In some implementations, a user of a userdevice provides the search query as an input, such as an input to asearch engine, prompt, text box, etc. In some implementations, thesearch query may be automatically generated based on informationassociated with the user device, such as sensor input or othercontextual based information. In an illustrative implementation, a userdevice or data processing system may generate a search query based on alocation of a user device. In some implementation, the data processingsystem receives a search query via an application program executing on auser device or a web page.

At stage 410, the method 400 includes identifying an entity of thesearch query and a confidence score. In some implementation, the methodincludes a data processing system (e.g., via a query reference module)identifying the entity. The method 400 may include accessing a datastructure having entity information. The method 400 may includeidentifying a confidence score for the entity that represents a semanticrelevancy of the entity to the search query. In some implementations,the data processing system identifies zero, one or many entities. Insome implementations, the data processing system identifies multipleinterpretations of a search query, where one or more of the multipleinterpretations include one or more entities.

At stage 415, the method 400 includes generating or identifying a querygraph with linked nodes. In some implementations, the method 400includes a data processing system (e.g., via the query reference module)generating the query graph. The query graph may represent a subset ofthe data structure. The subset may be a commercially relevant subset orother subset that facilitates content selection using entity properties.In some implementations, the search query may include multiple entities,in which case the query graph will include multiple entities andcorresponding links (e.g., subject, predicate, object triples for eachentity).

At stage 420, the method 400 includes retrieving or identifying one ormore content selection criteria graph for a content item that couldpotential match the search query graph. The content selection criteriagraph may be provided by a content provider and may be associated withone or more content items. The content selection criteria graph mayinclude one or more linked nodes.

The method 400 may include using a search infrastructure or module toidentify the potentially matching content selection criteria graphs. Insome implementations, the method 400 includes translating the searchquery graph into a flat data structure or list that includes the objectsof the search query graph, and retrieving some or all content selectioncriteria graphs that include the objects. Since the list of objects maynot include its corresponding topology in the search query, theidentified candidate content selection criteria graphs may or may notultimately match.

At step 425, the method 400 includes determining a match between thecontent selection criteria graph and the query graph. The method 400include a data processing system (e.g., via a matching module)determining the match. The method 400 may include stepping through eachnode of the content selection criteria graph to determine whether itmatches a corresponding node of the contents selection criteria graph.That is, the method 400 may include comparing the content selectioncriteria graph with the search query graph on a node-by-node basis todetermine whether the sequences or topologies match.

In some implementations, the method 400 includes determining whether theconfidence score exceeds a threshold, which may be predetermined, set bya content provider in the content selection criteria graph, ordynamically adjusted based on performance feedback.

At step 430, the method 400 includes selecting the content item as acandidate for display on the user device. The method 400 may include adata processing system providing a content item, content group, contentcampaign, or identifiers of same, to a content selector. These contentitems may enter an online content item auction which may determine,based on additional factors such as bids submitted by content providers,whether the content item will ultimately be provided to a user devicefor presentation.

FIG. 5 illustrates one implementation of a system 500 for selectingcontent via a computer network such as network 105. The system 500 andits components, such as a data processing system 520, may includehardware elements, such as one or more processors, logic devices, orcircuits. In some implementations, system 500 may include one orcomponents of system 100, or otherwise be designed, constructed orconfigured to include one or more functionality of system 100. In someimplementations, system 500 and system 100 may be the same system. Insome implementations, system 100 includes one or more component orfunctionality of system 500.

The system 500 can include at least one data processing system 520. Thedata processing system 520 can include at least one logic device such asa computing device having a processor to communicate via the network 105with the user device 110, the content publisher 115, and at least onecontent provider 125. The data processing system 520 can include one ormore components of data processing system 120, or otherwise be designed,constructed or configured to include one or more functionality of system100. In some implementations, data processing system 520 and dataprocessing system 120 may include the same components, be configured thesame, or be the same data processing system. In some implementations,data processing system 120 includes one or more component orfunctionality of system 520.

The data processing system 520 can include at least one an interfacemodule 535, at least one query reference module 540, at least onematching module 545, at least one lookup module 550, and at least onedatabase 555. The at least one an interface module 535, at least onequery reference module 540, at least one matching module 545, and atleast one lookup module 550 can each include at least one processingunit or other logic device such as programmable logic array engine, ormodule configured to communicate with the database 555. The at least oneinterface module 535, at least one query reference module 540, at leastone matching module 545, and at least lookup module 550 can include asingle component, or part of the data processing system 520 or dataprocessing system 120. In some implementations, the interface module 535and interface module 135

In some implementations, interface module 535 may include one orcomponents of interface module 135, or otherwise be designed,constructed or configured to include one or more functionality ofinterface module 135. In some implementations, interface module 535 andinterface module 135 may be the same interface module. In someimplementations, interface module 135 includes one or more component orfunctionality of interface module 535.

In some implementations, the data processing system 520 or interfacemodule 535 is configured to receives an indication to generate contentselection criteria based on target content. The interface module 535 mayreceive the target content. Receiving the target content may serve asthe indication to generate content selection criteria. In someimplementations, the data processing system 520 receives an indicationto generate content selection criteria and then provides a prompt toenter additional information that can be used to generate the contentselection criteria, such as the target content. In some implementations,the target content includes a query or a set of queries. In someimplementations, the target content includes an online document such asa web page. In some implementations, the target content includes a link,URL, or address of a web page or a data file that includes text, terms,keywords, or queries. The target content can refer to content that acontent provider is targeting. In an illustrative implementation, acontent provider may target a list of books such that an advertisementof the content provider is provided responsive to the data processingsystem 520 receiving a search query for a book associated with the listof book (e.g., by an author of the list of books).

In some implementations, the interface module 535 may provide agraphical user interface to a content provider 125 configured tofacilitate establishing, creating or modifying a content selectioncampaign or aspect thereof. In some implementations, a content provider125 may add or select content items and content selection criteria tofacilitate content selection and providing content items for display orother presentation via a user device 110.

In some implementations, the target content includes a query or sets ofqueries which are used to create content selection criteria. In anillustrative implementation, as shown in FIG. 6, the target content mayinclude a string such as “manhattan”. In other illustrations, the targetcontent may include a web page (e.g., a link, address, URL or otheridentifier of a web page), online document, data file, productinformation, or other data that can be processed by the data processingsystem 520 to identify one or more entities and create content selectioncriteria.

The data processing system 520 can include a query reference module 540.In some implementations, query reference module 540 may include one orcomponents of query reference module 140, or otherwise be designed,constructed or configured to include one or more functionality of queryreference module 140. In some implementations, query reference module540 and query reference module 140 may be the same query referencemodule. In some implementations, query reference module 140 includes oneor more component or functionality of query reference module 540.

The query reference module 540 can use the target content (e.g., targetcontent including the query or set of queries) provided via the contentprovider 125 to identify one or more entities of the target content andcreate, generate or identify a property of the entity. The property ofthe entity may include or be associated with a category, topology,hierarchical semantic structure, or a query graph. In someimplementations, the query reference module 540 identifies informationabout entities of the query using a data structure storing entityinformation. In some implementations, query reference module 540creates, identifies, or uses a query graph or data structure with entityinformation in a manner similar to query reference module 140. In someimplementations, the data processing system 520 receives a list ofqueries or parses a web page or other electronic document to identify alist of queries. The data processing system 520, using the list ofqueries, can identify the one or more entities of the one or morequeries. In some implementations, the data processing system 520identifies one or more interpretations of the query, where eachinterpretation may include zero, one or many entities. In someimplementations, the data processing system 520 identifies a confidencescore corresponding to the instance of the entity or interpretation.

The query reference module 540, as discussed in relation to queryreference module 140, accesses a data structure comprising entityinformation to annotate or otherwise associate information with theentity. The information may include properties or other relations of theentity. The data processing system 520 can determine that the targetcontent provided by the content provider 125 “mentions an X that isrelated to EntityID_Y via the relation /A/B/C”. In an illustrativeimplementations, where the target content (or query or set of queries)includes a list of books, the data processing system 520 determines that“the target content mentions a book that is related to Person_X via therelation /book/author/works_written”.

In some implementations, the data processing system 520 includes alookup module 550 designed and constructed to identify, access orotherwise obtain one or more templates based on the target contentprovided by the content provider 125. In some implementations, thelookup module 550 can parse a data structure stored in a memory (e.g.,database 555) to identify one or more templates.

The data processing system 520 can use the template to facilitateidentifying or creating content selection criteria based on the targetcontent provided by the content provider 125. The template may include atopology and a named variable. A named variable may include a variablethat can be named and assigned values. The values may refer to entityidentifiers, strings, symbols, etc. Named variable can correspond totypes or categories of entities. In an illustrative implementation,named variables may include, e.g., one or more of $Area, $Collection,$Directors, $Genres, etc. Named variables may be based on a query graphor entity data structure.

In an illustrative implementation, if the target content provided by thecontent provider 125 includes query “manhattan”, the query referencemodule 540 may annotate the query with a property or relation such as“location/location/containedby”. In some implementations, a property ofthe entity includes a second entity and a relation between the entityand the second entity. Based on this relation, the lookup module 550 canidentify one or more templates that include a topology (e.g., “querymentions entity /location/location/containedby”) and a named variable(e.g., $Area). In another illustrative implementation, the queryprovided by the content provider 125 may be “Eiffel tower”. The dataprocessing system 520 may annotate the query with one or relations orproperties such as “/location/location/containedby” and “in_collection”.Based on these relation types, the data processing system may identify atemplate “query mentions entity (in_collection $Collection AND/location/location/containedby $Area)” that includes two named variables$Collection and $Area.

In some implementations, the data processing system 520 identifies atemplate that corresponds to some or all the relations associated withone or more entity of the query provided by the content provider 125. Insome implementations, the data processing system 520 identifies atemplate for one or more interpretations of the query identified by thequery reference module 540. In some implementations, the templates maycorrespond to commercially relevant relations or properties of the oneor more entities of the query. In an illustrative implementation, thecontent provider 125 may indicate, or the data processing system 520 mayotherwise determine (e.g., based on predetermined information orhistorical analysis) that using one or more types of named variables(e.g., $Area or $Collection) to create content selection criteria mayresult in effective content selection (e.g., improved performance basedon click through rate or conversion rate).

In some implementations, an administrator of the data processing system520 may generate or store templates in a database that can be accessedor used by the data processing system 520 to create content selectioncriteria. The stored templates may include templates for one or morerelations and named variables. In some implementations, the database maystore a list of commercially relevant named variables and relations, andcombine them or otherwise use them to form a template in real-time(e.g., upon receiving a request to generate content selection criteriausing a query provided by a content provider). In an illustrativeimplementation, the database may include a relation/location/containedby and a named variable $Area associated with thatrelation. The database may also include a relation /in/_collection and anamed variable $Collection associated with that relation. The dataprocessing system 520, upon identifying information of an entityassociated with a query, may combine the two relations and namedvariables to form a template that includes both relations and namedvariables.

In some implementation, the data structure stored in database 555 mayinclude, for one or more relations, a corresponding named variable. Therelations and named variables may correspond to one or more templates.The one or more templates (or set of templates) stored in a datastructure of database 555 (or other memory accessible by data processingsystem 520) may be dynamically generated, modified, or updated based ona time interval or receiving an instruction to update the set oftemplates. In some implementations, a content provider 125 may provide aset of templates for use in creating content selection criteria forcontent items of the content provider 125. In some implementations, aset of templates may correspond to a type of content provider 125,vertical (e.g., automotive, travel, food, sports), market, audience,etc.

In some implementations, the lookup module 550 identifies or determinessemantic criteria associated with or corresponding to a named variable.In an illustrative implementation, a named variable $Area may correspondto, include, or otherwise be associated with or linked to instances ofthe named variable (which may include entities or unique entityidentifiers) such as “United States”, “California”, “New York City”,“Suffolk County”, etc. The lookup module 550 may identify the associatedinstances via parsing or otherwise analyzing a data structure providingstructured and detailed information about persons, places or thingsassociated with unique entity identifiers. In an illustrativeimplementation, the entity “United States” may include a property“/location” in the data structure, and the data processing system 520may determine, based on the property “/location”, that “United States”corresponds to named variable $Area. A semantic criteria may include thetopology identified in the template with an instance of one or morenamed variable.

In some implementations, the data processing system 520 identifiessemantic criteria that corresponds to annotated information of an entityof a search query provided by a content provider. In an illustrativeimplementation shown in FIG. 6, the query “manhattan” (602) mentions(604) entityID_10 “Manhattan” (610), which is related to entityID_11“United States of America” (620) via relation“location/location/containedby” (612). Thus, the data processing system520 may identify semantic criteria 628 that corresponds to the namedvariable $Area and matches the EntityID_11.

In some implementations, the semantic criteria identified based on thetemplate and the named variables match the query graph because thetemplate and named variables are selected to match the query graph. Inan illustrative implementation, a query graph may include topology withentities “query (mentions entity /location/location/containedby/m/EntityID_x AND mentions entity /common/topic/notable_type/m/EntityID_y)”, and a selected template may include topology and namedvariables “query (mentions entity /location/location/containedby $AreaAND mentions entity /common/topic/notable_type $Type)”, where at leastone instance of named variable $Area matches EntityID_x and at least oneinstance of named variable $Type matches EntityID_y. Thus, in someimplementations, by identifying a template with named variables thatmatches the query graph, and corresponding instances of the namedvariables, the semantic criteria based on the template and the instancesof the named variables are identified as matching the query graph.

In some implementations, the data processing system 520 includes amatching module 545 designed and constructed to identify or createcontent selection criteria based on named variables of a template andentities mentioned in target content (or query or set of queries)provided by a content provider 125. In some implementations, thematching module 545 maps the semantic criteria or entities of the namedvariable (628) to the entities of the query graph (e.g., 620, 622, 624)to determine whether one or more entities of the query graph match anentity of the named variable. In some implementations, it may not benecessary to perform this match because the semantic criteria identifiedby the lookup module 550 using the template, named variables, andinstances of the named variables is guaranteed to match the query graphbecause the template is already identified as matching the query graph.

In some implementations, upon determining a match, the data processingsystem 520 may provide, suggest or otherwise indicate that one or morematching entity may be used as content selection criteria. In someimplementations, the data processing system 520 may identify thematching entity (e.g., EntityID_11 “United States of America” 620) foruse as content selection criteria based on the received query. In someimplementations, the data processing system 120 may identify multiplematching semantic criteria and perform further processing or analysis toidentify one or more matching semantic criteria for use as contentselection criteria by the content provider.

In some implementations, the data processing system 520 identifies amatching semantic criteria on a node-by-node basis by stepping throughthe nodes and links as illustrated in FIG. 6. In this illustrativeimplementation, for the named variable $Area 642 to match, the contentof the named variable $Area (e.g., the semantic criteria 628) and thetopology or sequence (e.g., 630-640) is mapped or compared to the querygraph 600 as shown by matching lines 650.

In some implementations, where a template includes multiple namedvariables, the data processing system may identify a semantic criteriathat corresponds to the named variables individually. In someimplementations, the data processing system 520 identifies a semanticcriteria that corresponds to both of the named variables.

Upon identifying one or more matching semantic criteria 628 (e.g.,/m/EntityID_11 and /m/Entity_ID_13), the data processing system 520 mayfurther determine whether one or more of these semantic criteria may beeffective in content selection. In some implementations, the dataprocessing system 520 determines a metric indicative of theeffectiveness of the semantic criteria in content selection. The metricmay quantify how strongly the presence or absence of property A isassociated with the presence or absence of property B in a given corpus.In some implementations, the data processing system 520 determines ametric based on a statistical technique such as an odds ratio or a termfrequency-inverse document frequency. The data processing system 520 mayuse the statistical metric to select one or more of the matchingsemantic criteria to be candidate content selection criteria, andprovide or suggest these candidate content selection criteria to acontent provider for use in a content selection campaign.

In some implementations, the statistical metric may represent aninformation content of the matching semantic criteria (e.g., based on aterm frequency-inverse document frequency (“tf-idf”) where documentscorrespond to queries). In an illustrative implementation, if a newpiece of information is true for 90% of queries, then the new piece ofinformation may not be useful. The tf-idf may include a numericalstatistic that reflects how important a word is to a query in acollection or corpus of queries. The tf-idf value may increase (e.g.,proportionally) to the number of times a word appears in the corpus ofqueries, but may be offset by the frequency of the word in the corpus.

In an illustrative implementation, the target content provided by acontent provider may include two queries “Eiffel Tower” and “Big Ben”.Based on these queries, the data processing system 520 may identifymatching semantic criteria “building in Europe” and “building inFrance”. The data processing system 520 may further determine that thefirst matching semantic criteria matches both queries provided by thecontent provider 125, while the second matching semantic criteriamatches one of the queries provided by the content provider 125. Thus,the term frequency for “building in Europe” may be 2, while the termfrequency for “building in France” may be 1. The term frequency mayrefer to the number of queries provided by the content provider withwhich the semantic criteria matches.

The data processing system may further incorporate an inverse documentfrequency factor with the term frequency to determine the statisticalmetric to facilitate distinguishing between relevant and non-relevantinformation. In some implementations, inverse document frequency refersto inverse query frequency (e.g., document refers to query). The inversedocument frequency may be determined across a large corpus of historicalqueries. In some implementations, the data processing system 520 mayinclude or have access to a log of historical queries provided by one ormore content providers 125, and use that information to determine theinverse document frequency. The inverse document frequency (e.g.,inverse query frequency) is a measure of whether the query, term, phraseor semantic criteria is common or rare across a corpus or collection ofdocuments or queries (e.g., historical queries). In someimplementations, the data processing system 520 determines the inversedocument frequency by dividing the total number of documents (orqueries) by the number of documents (or queries) containing the term. Insome implementations, the data processing system may further take alogarithm of this quotient.

The data processing system 520 may then determine the odds ratio bytaking the product of the term frequency and the inverse documentfrequency. Further to the illustrative implementation above, the totalnumber of queries in the corpus may be 10, and the number of queriescomprising the first semantic criteria may be 5, while the number ofqueries comprising the second semantic criteria may be 2. Thus, theinverse document frequency for the first semantic criteria may be thelogarithm of 10 divided by 5, and the inverse document frequency for thesecond semantic criteria may be the logarithm of 10 divided by 2.Finally, the tf-idf, or odds ratio, may be the product of the termfrequency and the inverse document frequency. The tf-idf for the firstsemantic criteria may be 2*log(10/5) and the tf-idf may be 1*log(10/2),or 0.6 and 0.7, respectively. The data processing system may use theresulting statistical metric as a weight or score when determiningwhether to include one or more matching semantic criteria as contentselection criteria, or may provide the weight or score to a contentprovider 125 to facilitate selecting content selection criteria. In someimplementations, the data processing system ranks the matching semanticcriteria based on the statistical metric.

In some implementations, the data processing system 520 stores thecomputed statistical metric in a database. In some implementations, thedata processing system 520 associates or otherwise assigns or links thestatistical metric with a content selection criteria or contentcampaign. The data processing system 520 may determine the statisticalmetric, or one or more component thereof, in an offline process, basedon a time interval (e.g., periodic, daily, weekly, hourly), orresponsive to a request to determine a statistical metric. The dataprocessing system 520 may determine the statistical metric in real-time,such as when a content provider 125 requests additional contentselection criteria for a content selection campaign or provides a set ofqueries.

In some implementations, the data processing system 520 ranks thematching semantic criteria based on the determined statistical metric.The data processing system 520 may provide the highest ranking matchingsemantic criteria (e.g., top 3, top 5, top 10, etc.) as a suggestion forcandidate content selection criteria (e.g., the content provider mayselect one or more of the suggested candidate content selection criteriafor inclusion or use in a content selection campaign).

In some implementations, the data processing system 520 determineswhether a statistical metric of a matching semantic criteria satisfies athreshold (e.g., meets or exceeds an odds ratio threshold). Thethreshold may be predetermined by an administrator of the dataprocessing system 520, or provided by a content provider 125. In someimplementations, the threshold may be a dynamic threshold that isdetermined based on performance feedback (e.g., click through rate orconversion rate of content selection criteria and their correspondingodds ratio) in order to improve the effectiveness of a content selectioncampaign. In some implementations, the data processing system 520 mayuse a logistic regression model or other machine learning technique todetermine a threshold based on performance feedback.

In some implementations, upon identifying multiple named variables andsemantic criteria for the named variables, the data processing system520 may form a set of combinations of the semantic criteria and furtherdetermine a candidate content selection criteria by taking a Cartesianproduct of the sets (e.g., a Cartesian product of the statistical metricof the individual semantic criteria or instances of the named variable).In an illustrative implementation, the query received by a contentprovider may be “eiffel tower”. The data processing system may identifya template that includes a named variable for area and named variablefor collection as follows: “query mentions entity (in_collection$Collection AND /location/location/containedby $Area)”. The dataprocessing system 520 may identify three matching semantic criteria forthe named variable $Collection (e.g., “towers”, “buildings”, and“tourist attractions”) and three different matching criteria for thenamed variable $Area (e.g., “France”, “Paris”, and “Champ de Mars”). Thedata processing system 520 may further identify a statistical metric(e.g., an odds ratio or tf-idf) for each of the six matching semanticcriteria. Thereafter, in order to identify, select or suggest acandidate content selection criteria based on the template, the dataprocessing system 520 may identify all combinations of the threecollection-related semantic criteria with the three area-relatedsemantic criteria and determine a statistical metric for eachcombination (e.g., by multiplying the individual statistical metrics,taking an average of the individual statistical metrics, summing theindividual statistical metrics or otherwise manipulating the individualstatistical metrics to determine a combined statistical metric). In someimplementations, the data processing system 520 may use a Cartesianproduct of these two sets to determine a statistical metric for eachcombination. The data processing system 520 may then identify a contentselection criteria based on the combined odds ratios. In an illustrativeimplementations, the data processing system 520 may identify thefollowing candidate content selection criteria: “search queries thatmention towers in Paris”. Thus, the data processing system 520 mayidentify content selection criteria for templates with multiple namedvariables using Cartesian product of the statistical metric of thematching semantic criteria.

In some implementations, a content provider, upon receiving one or morecandidate content selection criteria, may select some or all of thecandidate content selection criteria. In some implementations, theselect content selection criteria may be affirmative content selectioncriteria or negative content selection criteria. Affirmative contentselection criteria may refer to selecting content items when a requestfor a content item is associated with the content selection criteria(e.g., an advertisement for a Mexican restaurant responsive to a usesearch query that mentions Mexican food). In some implementations, thecontent selection criteria may be used as negative content selectioncriteria such that content items of the content provider is blocked,prevented, prohibited, determined ineligible, or otherwise not selectedwhen a request for a content item is associated with the negativecriteria (e.g., block the advertisement for the Mexican restaurant frombeing provided responsive to a search query that mentions fast food).

In some implementations, the data processing system 520 provides anatural language description of the matching semantic criteria. The dataprocessing system 520 can provide the natural language description basedon properties of an entity, a query graph, template, and a display nameof an entity corresponding to an instance of a named variable of thetemplate. The data processing system 520 may provide, transmit orotherwise present the natural language description or rendering of thetemplate to the content provider via the network.

In some implementations of providing a natural language description, thetemplate may include “query mentions entity /film/film/directedby$Director” and a natural language template may include “Query mentions amovie directed by {Director}”. In some implementations, the dataprocessing system may account for grammatical structure in generatingthe natural language version of the template. In an illustrativeimplementation, the query may include “Alien”, and the data processingsystem may provide, in natural language, the following: “Query mentionsa movie directed by Ridley Scott”.

In some implementations, the data processing system 520 determines anestimated traffic flow for the candidate content selection criteria. Thedata processing system 520 may determine the estimated traffic flowbased on a log that includes historical queries that resulted in trafficbeing directed to a web page or domain. In an illustrativeimplementation, the data processing system 520 may parse a log of webpage visits and identify all visits that resulted from a querycorresponding to the candidate content selection criteria, and furtherdetermine that the candidate content selection criteria resulted in 1000web page views for a content provider over a time interval (e.g., a day,week, month, etc.).

FIG. 6 is an illustration of one implementation of creating contentselection criteria using entity properties. A data processing system mayreceive an indication of a target content such as the query “manhattan”(602) from a content provider (e.g., via a user interface, input textbox, parsing a landing page, or other online document) and identify,generate, create or otherwise use a query graph 600. A query graph mayrefer to a categorical or hierarchical structure of informationassociated with entities, such as properties of the entity. The querygraph may further include relations between entities. The dataprocessing system may identify a template 626 with a named variable andidentify matching semantic criteria 628 based on matching content andtopology.

To generate or identify the properties of the entity or query graph 600,the data processing system may determine that the query 602 (e.g.,target content) mentions 604 a first entity 606. The first entity may beidentified as an entity with a unique identifier /m/entityID_10, whichmay be rendered as a string “Manhattan”. The data processing system maydetermine an entity confidence score of 0.74 for entity 610. The dataprocessing system may further determine that the query 602 mentions 604entityID_10 which is related to 620, 622 and 624 via relations 612, 614,616 and 618. That is, the data processing system determines that thequery “manhattan” (602) mentions (604) a first entity (606) ofentityID_10 (610) with a confidence score (608), and the entityID_10(610) is related to: EntityID_11 “United States of America” (620) viathe relation “/location/location/containedby” (612) and the relation“/projections/simple/whole_part/part_of” (616); EntityID_12“Administrative Division” (622) via the relation“/common/topic/notable_types” (614); and EntityID_13 “New York” (624)via the relation “/location/location/containedby” (618).

The data processing system identifies a template 626 based on thereceived query 602. The template 626 may include “query mentions entity/location/location/containedby $Area”. In some implementations, thetemplate may include multiple named variables (e.g., $Area and$Collection). The data processing system may identify semantic criteriathat correspond to the named variable. Upon identifying the query graph,template, named variable, and potential semantic criteria, the dataprocessing system may determine whether the semantic criteria match thequery graph (e.g., via matching lines 650). The matching semanticcriteria 626 may match the query graph based on content and topology.That is, the data processing system may map, compare, or match thesequence or graph 642 that includes query 630 mentions 632 entity 636relation 640 and content of named variable 642 with the query graph 600to identify matching semantic criteria 628. In this illustrativeimplementations, the matching semantic criteria 628 for query manhattan602, based on content and topology, includes EntityID_11 “United Statesof America” and entityID_13 “New York” (628). The data processing mayselect one or more matching semantic criteria 628 as candidate contentselection criteria or perform further processing (e.g., using an oddsratio, performance metric, or other quality metric) to identifyeffective content selection criteria.

FIG. 7 is an illustration of one implementation of a method 700 ofselecting content via a computer network. The method 700 includes a dataprocessing system receiving an indication to generate content selectioncriteria based on target content (705). The method 700 can includeidentifying an entity of the target content and a property of the entity(710). The method 700 can include accessing a template corresponding tothe property of entity (715). The method 700 can include determiningsemantic criteria based on the template that matches the property of theentity (720). The method 700 can include selecting candidate contentselection criteria based on a statistical metric of each of the matchingsemantic criteria (730).

The method 700 includes a data processing system receiving an indicationto generate content selection criteria based on target content (705). Insome implementations, the data processing system receives an indicationof a query. An indication of a query may be received via a userinterface of the data processing from a device of a content provider.The indication of a query may include a string, text, characters,symbols, audio, multimedia etc. The indication of a query may alsoinclude an online document, data file, web page, landing web page of acontent provider.

In some implementations, the indication of a query may includehistorical search queries that resulted in traffic directed to a webpage of the content provider (e.g., users directed to a landing web pageof a content provider). In an illustrative implementations, a dataprocessing system may receive, obtain or store a log that includessearch queries that resulted in users being directed to a web page of acontent provider. In some implementations, the log may includeadditional information that may facilitate generating content selectioncriteria such as information associated with a previously viewed webpage or a selected content item. In some implementations, a contentprovider may provide this information via a batch process to the dataprocessing system.

The method 700 can include identifying an entity of the target contentand a property of the entity (710). In some implementations, the method700 can include identifying an entity of the query to generate a querygraph (or otherwise identify properties of an entity of the query). Insome implementations, a property of the entity includes a second entityand a relation between the entity and the second entity. A dataprocessing system (e.g., via a query reference module) may identifyzero, one or multiple entities of the query. The method 700 may includestoring the identified entities in a memory or using the identifiedentities for further processing.

In some implementations, the method 700 includes determining aconfidence score for the identified entity of the query, determiningwhether the confidence score satisfies a threshold, and obtaining,generating, creating or otherwise identifying a query graph using theentity based on the entity satisfying the threshold.

The method 700 can include accessing a template corresponding to theproperty of entity (715). In some implementations, where the dataprocessing system identifies multiple entities in the received query,the method 700 may include identifying a template that corresponds toone or more of the multiple entities. In some implementations, thetemplate includes one or more named variables and relations ortopologies. In some implementations, the template may include multiplenamed variable if it is determined that there are multiple commerciallyrelevant relations associated with the entity mentioned in the queryreceived from the content provider.

The method 700 can include determining semantic criteria based on thetemplate that matches the property of the entity (720). In someimplementations, the method 700 can include determining semanticcriteria based on the template that matches the query graph (orproperties of the entity of the query). In some implementations, themethod 700 includes determining semantic criteria corresponding to oneor more named variables of the template. In some implementations, thesemantic criteria may include instances of a named variable of thetemplate. In some implementations, the semantic criteria based on theinstances of the one or more named variables of the template match thequery graph. The method 700 may include identifying the semanticcriteria via a data structure comprising information about entities.

In some implementations, the method 700 can include identifying semanticcriteria that match the query graph or properties of the entity of thequery. In some implementations, a property of the entity includes asecond entity and a relation between the entity and the second entity.The method 700 may include matching the content and the topology of thesemantic criteria with the query graph on a node-by-node basis toidentifying matching semantic criteria.

The method 700 can include selecting candidate content selectioncriteria based on a statistical metric of each of the matching semanticcriteria (730). In some implementations, the method 700 can includeusing a statistical metric (e.g., tf-idf or odds ratio) of each of thematching semantic criteria to select candidate content selectioncriteria. In some implementations, where the template includes multiplenamed variables, the method may include determining a Cartesian productbased on the statistical metrics for semantic criteria corresponding tothe multiple named variables. The method 700 may include selecting topranking matching semantic criteria as candidate content selectioncriteria, or otherwise selecting some or all of the matching semanticcriteria using metrics such as performance metric or quality metric. Insome implementations, the statistical metric includes metrics based onclick through rate, conversion rate, cost-per-click, performancefeedback, etc.

FIG. 8 is a block diagram of a computing system 800 in accordance withan illustrative implementation. The computing system or computing device800 can be used to implement the system 100 or 500, content provider125, user device 110, content publisher 115, data processing system 120,at least one interface module 135 and 535, at least one query referencemodule 140 and 540, at least one matching module and 545, at leastcontent selector 150, at least one lookup module 550, and at least onesearch module. The computing system 800 includes a bus 805 or othercommunication component for communicating information and a processor810 or processing circuit coupled to the bus 805 for processinginformation. The computing system 800 can also include one or moreprocessors 810 or processing circuits coupled to the bus for processinginformation. The computing system 800 also includes main memory 815,such as a random access memory (RAM) or other dynamic storage device,coupled to the bus 805 for storing information, and instructions to beexecuted by the processor 810. Main memory 815 can also be used forstoring position information, temporary variables, or other intermediateinformation during execution of instructions by the processor 810. Thecomputing system 800 may further include a read only memory (ROM) 820 orother static storage device coupled to the bus 805 for storing staticinformation and instructions for the processor 810. A storage device825, such as a solid state device, magnetic disk or optical disk, iscoupled to the bus 805 for persistently storing information andinstructions.

The computing system 800 may be coupled via the bus 805 to a display835, such as a liquid crystal display, or active matrix display, fordisplaying information to a user. An input device 830, such as akeyboard including alphanumeric and other keys, may be coupled to thebus 805 for communicating information and command selections to theprocessor 810. In another implementation, the input device 830 has atouch screen display 835. The input device 830 can include a cursorcontrol, such as a mouse, a trackball, or cursor direction keys, forcommunicating direction information and command selections to theprocessor 810 and for controlling cursor movement on the display 835.

According to various implementations, the processes described herein canbe implemented by the computing system 800 in response to the processor810 executing an arrangement of instructions contained in main memory815. Such instructions can be read into main memory 815 from anothercomputer-readable medium, such as the storage device 825. Execution ofthe arrangement of instructions contained in main memory 815 causes thecomputing system 800 to perform the illustrative processes describedherein. One or more processors in a multi-processing arrangement mayalso be employed to execute the instructions contained in main memory815. In alternative implementations, hard-wired circuitry may be used inplace of or in combination with software instructions to effectillustrative implementations. Thus, implementations are not limited toany specific combination of hardware circuitry and software.

Although a computing system has been described in FIG. 8,implementations of the subject matter and the functional operationsdescribed in this specification can be implemented in other types ofdigital electronic circuitry, or in computer software, firmware, orhardware, including the structures disclosed in this specification andtheir structural equivalents, or in combinations of one or more of them.

Implementations of the subject matter and the operations described inthis specification can be implemented in digital electronic circuitry,or in computer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them. The subject matter described inthis specification can be implemented as one or more computer programs,i.e., one or more circuits of computer program instructions, encoded onone or more computer storage media for execution by, or to control theoperation of, data processing apparatus. Alternatively or in addition,the program instructions can be encoded on an artificially generatedpropagated signal, e.g., a machine-generated electrical, optical, orelectromagnetic signal that is generated to encode information fortransmission to suitable receiver apparatus for execution by a dataprocessing apparatus. A computer storage medium can be, or be includedin, a computer-readable storage device, a computer-readable storagesubstrate, a random or serial access memory array or device, or acombination of one or more of them. Moreover, while a computer storagemedium is not a propagated signal, a computer storage medium can be asource or destination of computer program instructions encoded in anartificially generated propagated signal. The computer storage mediumcan also be, or be included in, one or more separate components or media(e.g., multiple CDs, disks, or other storage devices).

The operations described in this specification can be performed by adata processing apparatus on data stored on one or morecomputer-readable storage devices or received from other sources.

The term “data processing apparatus” or “computing device” encompassesvarious apparatuses, devices, and machines for processing data,including without limitation a programmable processor, a computer, asystem on a chip, or multiple ones, or combinations of the foregoing.The apparatus can include special purpose logic circuitry, e.g., an FPGA(field programmable gate array) or an ASIC (application specificintegrated circuit). The apparatus can also include, in addition tohardware, code that creates an execution environment for the computerprogram in question, e.g., code that constitutes processor firmware, aprotocol stack, a database management system, an operating system, across-platform runtime environment, a virtual machine, or a combinationof one or more of them. The apparatus and execution environment canrealize various different computing model infrastructures, such as webservices, distributed computing and grid computing infrastructures.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a circuit, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor data (e.g., one or more scripts stored in a markup languagedocument), in a single file dedicated to the program in question, or inmultiple coordinated files (e.g., files that store one or more circuits,sub programs, or portions of code). A computer program can be deployedto be executed on one computer or on multiple computers that are locatedat one site or distributed across multiple sites and interconnected by acommunication network.

Processors suitable for the execution of a computer program include,without limitation, both general and special purpose microprocessors,and any one or more processors of any kind of digital computer.Generally, a processor will receive instructions and data from a readonly memory or a random access memory or both. The essential elements ofa computer are a processor for performing actions in accordance withinstructions and one or more memory devices for storing instructions anddata. Generally, a computer will also include, or be operatively coupledto receive data from or transfer data to, or both, one or more massstorage devices for storing data, e.g., magnetic, magneto optical disks,or optical disks. However, a computer need not have such devices.Moreover, a computer can be embedded in another device, e.g., a mobiletelephone, a personal digital assistant (PDA), a mobile audio or videoplayer, a game console, a Global Positioning System (GPS) receiver, or aportable storage device (e.g., a universal serial bus (USB) flashdrive), to name just a few. Devices suitable for storing computerprogram instructions and data include all forms of non-volatile memory,media and memory devices, including without limitation semiconductormemory devices, e.g., EPROM, EEPROM, and flash memory devices; magneticdisks, e.g., internal hard disks or removable disks; magneto opticaldisks; and CD ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, implementations of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; in mostimplementations, feedback provided to the user can be any form ofsensory feedback, e.g., visual feedback, auditory feedback, or tactilefeedback; and input from the user can be received in any form, includingacoustic, speech, or tactile input.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinventions or of what may be claimed, but rather as descriptions offeatures specific to particular implementations of particularinventions. Certain features described in this specification in thecontext of separate implementations can also be implemented incombination in a single implementation. Conversely, various featuresdescribed in the context of a single implementation can also beimplemented in multiple implementations separately or in any suitablesubcombination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the implementations described above should not beunderstood as requiring such separation in all implementations, and itshould be understood that the described program components and systemscan generally be integrated in a single software product or packagedinto multiple software products.

References to “or” may be construed as inclusive so that any termsdescribed using “or” may indicate any of a single, more than one, andall of the described terms.

As used herein in the specification and in the claims, the phrase “atleast one,” in reference to a list of one or more elements, should beunderstood to mean at least one element selected from any one or more ofthe elements in the list of elements, but not necessarily including atleast one of each and every element specifically listed within the listof elements and not excluding any combinations of elements in the listof elements. This definition also allows that elements may optionally bepresent other than the elements specifically identified within the listof elements to which the phrase “at least one” refers, whether relatedor unrelated to those elements specifically identified. Thus, “at leastone of A and B” (or, equivalently, “at least one of A or B,” or,equivalently “at least one of A and/or B”) can refer, in oneimplementation, to at least one, optionally including more than one, A,with no B present (and optionally including elements other than B); inanother implementation, to at least one, optionally including more thanone, B, with no A present (and optionally including elements other thanA); in yet another implementation, to at least one, optionally includingmore than one, A, and at least one, optionally including more than one,B (and optionally including other elements); etc.

Thus, particular implementations of the subject matter have beendescribed. Other implementations are within the scope of the followingclaims. In some cases, the actions recited in the claims can beperformed in a different order and still achieve desirable results. Inaddition, the processes depicted in the accompanying figures do notnecessarily require the particular order shown, or sequential order, toachieve desirable results. In certain implementations, multitasking andparallel processing may be advantageous.

What is claimed is:
 1. A method comprising: receiving, using one or moreprocessors, a search query; identifying, using the one or moreprocessors, a plurality of entities referred to in the search query,wherein each of the entities is associated with an entity ID uniquelyassigned to the particular entity; generating, using the search query, asearch query graph that comprises: a) the search query; b) for each ofthe plurality of entities, the entity ID uniquely assigned to theparticular entity; c) for each of the plurality of entities, one or moreproperties of the particular entity, wherein the one or more propertiescomprise one or more other entities; and d) one or more relationshipconnections between the plurality of entities and the one or moreproperties; based on the search query, identifying one or more candidatecontent selection criteria graphs from a database using the searchquery, wherein each of the candidate content selection criteria graphsrelates to one or more of the identified entities referred to in thesearch query, wherein the one or more candidate content selectioncriteria graphs are associated with one or more content items;identifying, from the one or more candidate content selection criteriagraphs, a particular content selection criteria graph that matches thesearch query graph based, at least in part, on a similarity between theone or more entities, one or more properties, and one or morerelationship connections of the search query graph and the particularcontent selection criteria graph, wherein identifying a particularcontent selection criteria graph comprises comparing entity confidencescores of the search query graph with threshold confidence scores of theone or more candidate content selection criteria graphs; identifyingcontent items associated with the particular content selection criteriagraph that are distributed using the particular content selectioncriteria graph; and providing, for display on a computing device thatsubmitted the search query, one or more of the identified content items.2. The method of claim 1, wherein identifying a particular contentselection criteria graph comprises performing a node-by-node comparisonof the search query graph with the one or more candidate contentselection criteria graphs.
 3. The method of claim 2, wherein performinga node-by-node comparison comprises determining whether a topology ofthe search query graph matches a topology of one or more candidatecontent selection criteria graphs.
 4. The method of claim 1, whereinidentifying a particular content selection criteria graph comprisescomparing a topology of the search query graph with topologies of theone or more candidate content selection criteria graphs.
 5. The methodof claim 1, further comprising conducting an auction using bidssubmitted by content providers of the identified content items.
 6. Themethod of claim 1, wherein generating the search query graph comprisesaccessing a data structure having entity information and identifying oneor more properties of the plurality of entities from the entityinformation.
 7. The method of claim 1, wherein identifying one or morecandidate content selection criteria graphs from the database using thesearch query comprises identifying semantic criteria related to theplurality of entities.
 8. The method of claim 7, wherein the semanticcriteria corresponds to annotated information of an entity of a searchquery provided by a content provider.
 9. The method of claim 7, furthercomprising: after identifying the semantic criteria, determining theeffectiveness of the semantic criteria in content selection using ametric indicative of the effectiveness of the semantic criteria.
 10. Themethod of claim 9, wherein the metric is determined based on a termfrequency-inverse document frequency.
 11. The method of claim 9, whereinthe metric is determined based on an odds ratio.
 12. A systemcomprising: one or more computer processors; and one or morenon-transitory computer readable devices that include instructions that,when executed by the one or more computer processors, causes theprocessors to perform operations, the operations comprising: receiving,using the one or more processors, a search query; identifying, using theone or more processors, a plurality of entities referred to in thesearch query, wherein each of the entities is associated with an entityID uniquely assigned to the particular entity; generating, using thesearch query, a search query graph that comprises: a) the search query;b) for each of the plurality of entities, the entity ID uniquelyassigned to the particular entity; c) for each of the plurality ofentities, one or more properties of the particular entity, wherein theone or more properties comprise one or more other entities; and d) oneor more relationship connections between the plurality of entities andthe one or more properties; based on the search query, identifying oneor more candidate content selection criteria graphs from a databaseusing the search query, wherein each of the candidate content selectioncriteria graphs relates to one or more of the identified entitiesreferred to in the search query, wherein the one or more candidatecontent selection criteria graphs are associated with one or morecontent items; identifying, from the one or more candidate contentselection criteria graphs, a particular content selection criteria graphthat matches the search query graph based, at least in part, on asimilarity between the one or more entities, one or more properties, andone or more relationship connections of the search query graph and theparticular content selection criteria graph, wherein identifying aparticular content selection criteria graph comprises comparing entityconfidence scores of the search query graph with threshold confidencescores of the one or more candidate content selection criteria graphs;identifying content items associated with the particular contentselection criteria graph that are distributed using the particularcontent selection criteria graph; and providing, for display on acomputing device that submitted the search query, one or more of theidentified content items.
 13. The system of claim 12, whereinidentifying a particular content selection criteria graph comprisesperforming a node-by-node comparison of the search query graph with theone or more candidate content selection criteria graphs.
 14. The systemof claim 13, wherein performing a node-by-node comparison comprisesdetermining whether a topology of the search query graph matches atopology of one or more candidate content selection criteria graphs. 15.The system of claim 12, wherein identifying a particular contentselection criteria graph comprises comparing a topology of the searchquery graph with topologies of the one or more candidate contentselection criteria graphs.
 16. The system of claim 12, the operationsfurther comprising conducting an auction using bids submitted by contentproviders of the identified content items.
 17. The system of claim 12,wherein identifying one or more candidate content selection criteriagraphs from the database using the search query comprises identifyingsemantic criteria related to the plurality of entities.
 18. Anon-transitory computer-readable medium comprising processor executableinstructions to select content via a computer network, the instructionsfurther comprising instructions to: receiving a search query;identifying a plurality of entities referred to in the search query,wherein each of the entities is associated with an entity ID uniquelyassigned to the particular entity; generating, using the search query, asearch query graph that comprises: a) the search query; b) for each ofthe plurality of entities, the entity ID uniquely assigned to theparticular entity; c) for each of the plurality of entities, one or moreproperties of the particular entity, wherein the one or more propertiescomprise one or more other entities; and d) one or more relationshipconnections between the plurality of entities and the one or moreproperties; based on the search query, identifying one or more candidatecontent selection criteria graphs from a database using the searchquery, wherein each of the candidate content selection criteria graphsrelates to one or more of the identified entities referred to in thesearch query, wherein the one or more candidate content selectioncriteria graphs are associated with one or more content items;identifying, from the one or more candidate content selection criteriagraphs, a particular content selection criteria graph that matches thesearch query graph based, at least in part, on a similarity between theone or more entities, one or more properties, and one or morerelationship connections of the search query graph and the particularcontent selection criteria graph, wherein identifying a particularcontent selection criteria graph comprises comparing entity confidencescores of the search query graph with threshold confidence scores of theone or more candidate content selection criteria graphs; identifyingcontent items associated with the particular content selection criteriagraph that are distributed using the particular content selectioncriteria graph; and providing, for display on a computing device thatsubmitted the search query, one or more of the identified content items.